Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement flag metadata #3196

Merged
merged 14 commits into from
Jul 25, 2024
Merged

Conversation

kvnhmn
Copy link
Contributor

@kvnhmn kvnhmn commented Jun 19, 2024

Re: #2682 (only Backend implementation, no UI changes)

@kvnhmn kvnhmn requested a review from a team as a code owner June 19, 2024 13:47
Signed-off-by: Kevin Hermann <[email protected]>
Signed-off-by: Kevin Hermann <[email protected]>
Signed-off-by: Kevin Hermann <[email protected]>
Signed-off-by: Kevin Hermann <[email protected]>
@kvnhmn kvnhmn force-pushed the implement-flag-metadata branch from 23a941d to b773028 Compare June 19, 2024 13:50
Copy link
Member

@GeorgeMac GeorgeMac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @kvnhmn 🙌

This is amazing, thank you for taking a run at this.
We would love to help to get this over the line.
The bulk of the PR looks great 🙇

One thing that would help us would be to leverage the protobuf struct type (instead of the string type).
Really, we would like our old attachment to leverage that too, but its tricky with backwards compat and protobuf.
I added a comment in the review that gives some examples of where we use it in the auth storage layer for metadata there.

Another thing that would help is to update the declarative format.
You can find the definition for Flag here:

type Flag struct {
Key string `yaml:"key,omitempty" json:"key,omitempty"`
Name string `yaml:"name,omitempty" json:"name,omitempty"`
Type string `yaml:"type,omitempty" json:"type,omitempty"`
Description string `yaml:"description,omitempty" json:"description,omitempty"`
Enabled bool `yaml:"enabled" json:"enabled"`
Variants []*Variant `yaml:"variants,omitempty" json:"variants,omitempty"`
Rules []*Rule `yaml:"rules,omitempty" json:"rules,omitempty"`
Rollouts []*Rollout `yaml:"rollouts,omitempty" json:"rollouts,omitempty"`
}

Having Metadata here will allow folks to write it in yaml when either using a declarative backend or importing/exporting via the API.
This will need updating also in import and export:

  • import
    req := &flipt.CreateFlagRequest{
    Key: f.Key,
    Name: f.Name,
    Description: f.Description,
    Enabled: f.Enabled,
    NamespaceKey: namespace,
    }
  • export
    flag := &Flag{
    Key: f.Key,
    Name: f.Name,
    Type: f.Type.String(),
    Description: f.Description,
    Enabled: f.Enabled,
    }

Lastly, would be awesome to get at-least a happy path case in the integration tests.
But also, don't want to overload you. This could also be something we do in a subsequent PR.

Thanks for contributing 🙌

rpc/flipt/flipt.proto Outdated Show resolved Hide resolved
Copy link

codecov bot commented Jun 19, 2024

Codecov Report

Attention: Patch coverage is 89.47368% with 4 lines in your changes missing coverage. Please review.

Project coverage is 64.25%. Comparing base (140b023) to head (d84f66d).

Files Patch % Lines
internal/ext/importer.go 71.42% 1 Missing and 1 partial ⚠️
internal/storage/sql/common/flag.go 92.30% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3196      +/-   ##
==========================================
+ Coverage   64.19%   64.25%   +0.06%     
==========================================
  Files         168      168              
  Lines       13559    13582      +23     
==========================================
+ Hits         8704     8727      +23     
  Misses       4186     4186              
  Partials      669      669              
Flag Coverage Δ
unittests 64.25% <89.47%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@GeorgeMac GeorgeMac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making these changes @kvnhmn . It is looking great.

Unsure if you can see the failures in CI, but thought I would shine a light on some of them.

One of our test suites is seeing a non-nil pointer to an empty metadata:

         	            	--- Expected
        	            	+++ Actual
        	            	@@ -16779,3 +16779,16 @@
        	            	  Type: (flipt.FlagType) 0,
        	            	- Metadata: (*structpb.Struct)(<nil>)
        	            	+ Metadata: (*structpb.Struct)({
        	            	+  state: (impl.MessageState) {
        	            	+   NoUnkeyedLiterals: (pragma.NoUnkeyedLiterals) {
        	            	+   },
        	            	+   DoNotCompare: (pragma.DoNotCompare) {
        	            	+   },
        	            	+   DoNotCopy: (pragma.DoNotCopy) {
        	            	+   },
        	            	+   atomicMessageInfo: (*impl.MessageInfo)(<nil>)
        	            	+  },
        	            	+  sizeCache: (int32) 0,
        	            	+  unknownFields: ([]uint8) <nil>,
        	            	+  Fields: (map[string]*structpb.Value) <nil>
        	            	+ })
        	            	 })

Which, I think is fine for the API to return this, but either the API needs to change or that test case needs to be updated to ensure it is asserting a non-nil empty struct.

The other thing I am noticing is that the AFTER keyword on ALTER appears to not be supported by all of the relational DB backends.
What is weird is that the SQLite docs says it does not support it, but also, it doesn't fail on migration. Whereas e.g. Postgres fails explicitly during the migrations:

 35  : [27.4s] | Error: running migrations: migration failed: syntax error at or near "AFTER" (column 44) in line 1: ALTER TABLE flags ADD COLUMN metadata JSON AFTER enabled;
35  : [27.4s] |  (details: ERROR: syntax error at or near "AFTER" (SQLSTATE 42601))

We might need to drop that from the relevant migrations. It seems like maybe MySQL supports it, but not postgres and potentially SQLite just ignores it (so no harm I guess).

Signed-off-by: Kevin Hermann <[email protected]>
@kvnhmn
Copy link
Contributor Author

kvnhmn commented Jun 30, 2024

Thanks for making these changes @kvnhmn . It is looking great.

Unsure if you can see the failures in CI, but thought I would shine a light on some of them.

One of our test suites is seeing a non-nil pointer to an empty metadata:

         	            	--- Expected
        	            	+++ Actual
        	            	@@ -16779,3 +16779,16 @@
        	            	  Type: (flipt.FlagType) 0,
        	            	- Metadata: (*structpb.Struct)(<nil>)
        	            	+ Metadata: (*structpb.Struct)({
        	            	+  state: (impl.MessageState) {
        	            	+   NoUnkeyedLiterals: (pragma.NoUnkeyedLiterals) {
        	            	+   },
        	            	+   DoNotCompare: (pragma.DoNotCompare) {
        	            	+   },
        	            	+   DoNotCopy: (pragma.DoNotCopy) {
        	            	+   },
        	            	+   atomicMessageInfo: (*impl.MessageInfo)(<nil>)
        	            	+  },
        	            	+  sizeCache: (int32) 0,
        	            	+  unknownFields: ([]uint8) <nil>,
        	            	+  Fields: (map[string]*structpb.Value) <nil>
        	            	+ })
        	            	 })

Which, I think is fine for the API to return this, but either the API needs to change or that test case needs to be updated to ensure it is asserting a non-nil empty struct.

The other thing I am noticing is that the AFTER keyword on ALTER appears to not be supported by all of the relational DB backends. What is weird is that the SQLite docs says it does not support it, but also, it doesn't fail on migration. Whereas e.g. Postgres fails explicitly during the migrations:

 35  : [27.4s] | Error: running migrations: migration failed: syntax error at or near "AFTER" (column 44) in line 1: ALTER TABLE flags ADD COLUMN metadata JSON AFTER enabled;
35  : [27.4s] |  (details: ERROR: syntax error at or near "AFTER" (SQLSTATE 42601))

We might need to drop that from the relevant migrations. It seems like maybe MySQL supports it, but not postgres and potentially SQLite just ignores it (so no harm I guess).

Thanks for your tip @GerogeMac.

I have extended a non-nil empty struct in the integration tests. I also removed it from all migrations, except for the one for MySql, for sqlite I also removed it to be on the safe side, hope that's okay. Also i extended the DEVELOPMENT.md, cause i missed more informations regarding unit-/integration-tests

rpc/flipt/flipt.proto Outdated Show resolved Hide resolved
Copy link
Collaborator

@markphelps markphelps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking great! commented on failing ci tests

internal/ext/exporter.go Outdated Show resolved Hide resolved
internal/ext/common.go Outdated Show resolved Hide resolved
@erka erka force-pushed the implement-flag-metadata branch 4 times, most recently from a7ad122 to 5698077 Compare July 24, 2024 19:21
@erka erka force-pushed the implement-flag-metadata branch from 5698077 to cf2e5e3 Compare July 24, 2024 20:05
@erka erka requested review from markphelps and GeorgeMac July 24, 2024 20:18
Copy link
Collaborator

@markphelps markphelps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thank you both @kvnhmn and @erka !

@markphelps
Copy link
Collaborator

@all-contributors please add @kvnhmn for code

Copy link
Contributor

@markphelps

I've put up a pull request to add @kvnhmn! 🎉

@markphelps markphelps added the automerge Used by Kodiak bot to automerge PRs label Jul 25, 2024
@markphelps markphelps merged commit 489e492 into flipt-io:main Jul 25, 2024
35 checks passed
@markphelps markphelps mentioned this pull request Jul 25, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Used by Kodiak bot to automerge PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants